Reverse Engineering Airbnb's Search Ranking Algorithm

Project Overview

As a part-time property manager with a portfolio of rental properties in Cape Town, I wanted to understand what factors influence search rankings on Airbnb. Higher rankings mean more visibility, more bookings, and ultimately more revenue.

Airbnb doesn't publish their ranking algorithm, so I built a data pipeline to scrape listing data at scale, store it in a relational database, and perform statistical correlation analysis to reverse-engineer the factors that matter most.

The Data Pipeline

Since Airbnb doesn't provide a public API, I built a custom web scraping solution to gather the data needed for analysis:

Target Data Points

Each Airbnb listing displays surface-level information like price and reviews, but much richer data is hidden in the page's underlying JSON structure. I identified the key fields to extract:

Data extracted from each listing

Building the Scrapy Spider

I built a Python web spider using the Scrapy framework to systematically crawl Airbnb's search results. The spider was configured to target listings accommodating 6+ guests within specific GPS coordinates covering greater Cape Town.

airbnb_spider.py

The spider constructs URLs dynamically, appending search parameters (guest count, GPS boundaries) and iterating through all result pages automatically.

Parsing Hidden JSON Data

The real value came from extracting data embedded in each listing's HTML as JSON. This included fields not visible on the surface - satisfaction scores, response rates, calendar update frequency, and 30+ other variables.

parse_listing.py

Database Architecture

The scraped data was normalised into a relational structure with five interconnected tables, enabling flexible analysis across multiple dimensions:

Entity Relationship Diagram

The central Scrapy Data table contained the detailed listing information, joined to amenity codes (with a separate lookup table for descriptions), suburb data for location analysis, and a gender classification table to test whether host gender correlated with rankings.

Key Findings

Guest Satisfaction: The Dominant Factor

The most striking finding was the clear linear relationship between guest satisfaction scores and search ranking. Page 1 listings averaged 83.7% satisfaction, declining steadily to 42.9% on page 17.

Average Guest Satisfaction by Search Result Page

Clear inverse correlation: as page number increases, average guest satisfaction decreases.

Top 5 Ranking Factors

Statistical correlation analysis revealed the five factors most strongly associated with higher search rankings:

Factors 1-5: Strongest Correlation with Search Rank

Factors 6-10

Factors 6-10: Secondary Correlation Factors

Notable findings from the secondary factors include the importance of Instant Book (0.844 correlation), review count (0.828), and wishlist saves (0.819).

The Complete Correlation Table

All 27 factors tested, ranked by their correlation coefficient with search page position:

All Factors Ranked by Correlation to Search Position

Green = strong positive correlation with ranking. Red = weak or no correlation. Coefficients converted to absolute values for easier interpretation.

Detecting Algorithm Changes Over Time

By comparing data from February 2016 and February 2017, I was able to detect a significant shift in how Airbnb treats Instant Book listings:

Instant Book Correlation: 2016 vs 2017

This aligned with Airbnb's September 2016 announcement that they would "accelerate the use of Instant Book" to reach one million listings by January 2017, partly as a measure to reduce discrimination in the booking process.

Amenity Correlation Analysis

I also analysed which amenities correlated most strongly with higher search rankings:

Amenities Ranked by Correlation to Search Position

Red highlighted rows indicate "Business Ready" required amenities - notably these cluster in the top half of correlations.

Interestingly, "Business Ready" required amenities (iron, laptop workspace, hangers, hair dryer, essentials) all showed strong correlation with better rankings. Meanwhile, TV and Cable TV showed almost no correlation - perhaps less important for properties hosting 6+ guests.

Actionable Recommendations

Based on the correlation analysis, here are the most impactful actions a host can take to improve search ranking:

Project Impact

This research project delivered practical insights that I applied directly to the properties I manage, and it laid the groundwork for larger-scale research collaborations with professional property management companies.

The project demonstrated that with the right technical approach - web scraping, data engineering, and statistical analysis - it's possible to reverse-engineer opaque algorithms and extract actionable business intelligence.